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ATEC’s  Approach  to  Integrated  Testing 
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Integrated  testing  is  not  new  to  anyone  in  the  acquisition/testing  field.  When  asking  members  of 
the  Service  testing  communities  if  they  are  doing  integrated  testing,  the  reply  is  almost  always 
going  to  be  affirmative.  With  that  said,  what  do  they  mean  by  integrated  testing?  Historically, 
different  people  have  offered  different  definitions. 

Keywords:  ATEC  System  Team;  Ballistic  Missile  Defense  System;  combined  DT/OT; 
coordination;  integrated  DT/OT;  planning  testing  strategy. 


In  an  effort  to  provide  a  common  point  of 
reference,  the  Office  of  the  Secretary  of 
Defense  (OSD)  provided  a  definition  of 
integrated  testing  to  the  Services  in  2008. 
They  defined  it  as 

". . .  the  collaborative  planning  and  collaborative 
execution  of  test  phases  and  events  to  provide 
shared  data  in  support  of  independent  analysis, 
evaluation  and  reporting  by  all  stakeholders 
particularly  the  developmental  (both  contractor 
and  government)  and  operational  test  and 
evaluation  communities.  ”  (OSD,  2008) 

U.S.  Army  Test  and  Evaluation  Command  (ATEC) 
has  further  refined  this  definition  to  obtain  a  better 
understanding  of  where  we  can  go  beyond  integrated 
testing.  When  talking  about  integrating  testing,  ATEC 
uses  the  terms  “Integrated”  Developmental  Test  and 
Operational  Test  (DT/OT)  as  well  as  “Combined”  DT/ 
OT.  Integrated  DT/OT,  a  special  case  of  a  Combined 
DT/OT,  is  a  single-phased  event  that  generates  data  to 
address  developmental  and  operational  issues  simulta¬ 
neously  under  operational  conditions.  The  execution 
strategy  for  this  event  is  based  on  the  requirements  of 
the  program.  Combined  DT/OT  is  a  single  event  that 
produces  data  to  answer  developmental  and  operational 
system  issues.  A  Combined  DT/OT  is  usually  con¬ 
ducted  as  a  series  of  distinct  DT  and  OT  phases  at  a 
single  location  using  the  same  test  items.  For  the  case  in 
which  a  single  phase  can  be  used  to  simultaneously  meet 
developmental  and  operational  issues,  this  testing  will  be 
referred  to  as  an  Integrated  DT/OT.  Combined  DT/ 
OT  and  Integrated  DT/OT  are  encouraged  to  achieve 
time,  cost,  and  resource  savings.  However,  they  must 
not  compromise  DT  and  OT  objectives  in  accordance 
with  the  Defense  Acquisition  Guidebook. 


Roadmap 

ATEC  has  the  distinct  advantage  among  the  Service 
Operational  Test  Agencies  (OTAs)  of  having  the 
developmental  testers,  operational  testers,  and  the 
system  evaluators  all  organized  under  one  command. 
ATEC  utilizes  a  highly  effective  team  structure  for 
each  evaluated  system.  This  team  structure  is  labeled 
the  ATEC  System  Team  (AST).  The  team  is 
composed  of  personnel  from  each  subordinate  com¬ 
mand  activity  (SCA)  within  ATEC.  The  core 
members  include  the  developmental  tester,  operational 
tester,  and  the  evaluator.  The  AST  has  a  Chair  who  is 
the  command’s  lead  person  on  that  system. 

The  AST  plans,  manages,  and  coordinates  test  and 
evaluation  (T&E)  for  assigned  systems.  An  AST  is 
established  upon  notification  of  a  new  requirement 
from  any  source  or  receipt  of  justification  for  new 
requirements.  The  office  of  the  ATEC  Deputy  Chief 
of  Staff  for  Operations  coordinates  the  formation  of 
the  ASTs  across  ATEC. 

The  AST  identifies  and  resolves  system  T&E  issues 
and,  when  necessary,  elevates  unresolved  issues  up 
through  the  chain  of  command.  The  AST  presents  a 
single  coordinated  ATEC  position  at  the  T&E 
Working-level  Integrated  Product  Team  (T&E 
WIPT)  meetings.  The  AST  has  many  responsibilities. 
For  example,  the  AST  is  responsible  for  the  synchro¬ 
nization  and  integration  of  all  ATEC  efforts  for  the 
assigned  system;  speaks  with  one  voice  when  interact¬ 
ing  with  organizations  outside  ATEC;  reviews  all 
requirement  documents,  request  for  proposals,  and 
statement  of  works.  The  AST  Chair  leads  aU  AST 
activities. 

The  AST  members  (both  testers  and  evaluators)  are 
involved  early  in  the  T&E  planning  activities  in  order 
to  offer  their  expertise  and  begin  identifying  resource 
requirements.  The  AST  ensures  the  T&E  strategy  is 
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aligned  with  and  supports  the  approved  acquisition 
strategy,  so  that  adequate,  risk-reducing,  nonredundant 
gathering  of  T6cE  information  is  provided  to  support 
decision  events. 

The  evaluation  strategy  typically  utilizes  ground  test 
activities,  where  appropriate,  to  include  hardware-in- 
the-loop  simulation,  prior  to  conducting  full-up 
system-level  testing  in  realistic  environments.  The 
required  technical  progress  includes  reliahility,  desired 
capabilities,  and  satisfaction  of  critical  operational 
issues  and  criteria  to  mitigate  technical  and  manufac¬ 
turing  risks.  This  wiU  increase  the  likelihood  of 
operational  testing  and  evaluation  (OTScE)  success 
by  testing  in  the  most  realistic  environment  possible.  In 
addition,  the  AST  will  assess  system-of-systems 
command,  control,  communications,  computers,  intel¬ 
ligence,  surveillance,  and  reconnaissance  prior  to 
operational  testing  to  ensure  that  interoperability 
under  representative  load  conditions  will  represent 
stressed  OT  scenarios. 

The  AST  must  keep  ATEC  and  the  SCA  leadership 
informed  by  conducting  a  series  of  management 
reviews  of  planning,  progress,  and  results;  continually 
updating  program  status  and  milestones  in  the  ATEC 
Decision  Support  System. 

Methods/processes 

Planning  for  T&E  begins  at  the  earliest  stages  of  the 
development  of  user  needs,  science  and  technology, 
system  requirements,  development,  and  acquisition 
processes.  Evaluators  on  the  AST  participate  in  the 
Integrated  Concept  Development  Team  review  of  the 
Initial  Capabilities  Document  when  a  new  system  or 
new  technology  is  being  considered  for  development. 

By  policy,  every  T6cE  strategy  wiU  coUaboratively 
plan  all  testing  and  Modeling  and  Simulation  (M6cS) 
activities  as  an  efficient  continuum  to  include  appro¬ 
priate  use  of  integrated/combined  testing.  Both 
developmental  and  operational  testers,  in  concert  with 
the  system  evaluator,  assist  the  Materiel  Developer 
(MATDEV)  and  Combat  Developer  in  credible  events 
as  appropriate  to  the  program.  Early  involvement  can 
significantly  reduce  test  time  and  cost  through 
comparative  analysis,  data  sharing,  and  use  of  aU 
credible  data  sources. 

The  primary  purpose  of  the  T&E  WIPT  is  to 
optimize  the  use  of  appropriate  T&E  expertise,  test 
assets,  targets,  instrumentation,  facilities,  simulations, 
and  models  to  achieve  test  integration,  thereby 
reducing  costs  to  the  Army  and  decreasing  acquisition 
cycle  time.  The  T&E  WIPT  supports  the  collaborative 
T&E  strategy,  resolves  issues,  and  assists  the  Program 
Manager  (PM)/MATDEV  in  developing  and  coordi¬ 
nating  the  Test  and  Evaluation  Master  Plan. 


Expectations 

What  is  expected  by  the  overall  acquisition  com¬ 
munity  with  the  increase  of  integrated  testing? 
Program  Executive  Officer  (PEO)/PMs  expect  to  save 
time  and  money.  Because  of  that,  integrated  testing  is 
very  attractive  to  them.  However,  with  this  type  of 
integrated  DT/OT,  it  is  very  unclear  (from  the  tester’s 
perspective)  who  is  in  charge  of  the  planning, 
execution,  and  reporting. 

Current  OSD  guidance  is  that  developmental  and 
operational  test  activities  shall  be  integrated  and 
seamless  throughout  the  system’s  life  cycle.  This 
guidance  is  understood,  but  very  difficult  for  test 
agencies  to  implement  due  to  the  lack  of  resourcing. 
OSD  expects  better  results  from  the  systems  acquisi¬ 
tion  as  demonstrated  via  higher  pass  rate  in  initial 
OT&Es. 

Challenges 

•  Approval  of  plans — Director  OT&E  approves 
operational  test  plans  but  they  only  provide  advice 
on  developmental  testing.  With  an  integrated 
DT/OT,  who  would  approve  the  plans? 

•  Changing  cultures — Traditional  ways  of  plan¬ 
ning,  executing,  and  reporting  test  events  must 
change  into  a  more  cohesive  and  cooperative 
process.  How  do  we  decide  who  is  in  charge  of 
each  test  event? 

•  Competition  for  services  between  the  test  cen¬ 
ters — If  both  DT  and  OT  assets  are  being  used 
for  test  events,  wiU  there  be  enough  assets  to 
fulfill  DT-specific  or  OT-specific  events? 

•  Maintain  the  “independent”  evaluator — ^Will  the 
evaluator  be  able  to  remain  independent  if  they 
are  integrated  into  the  developmental  cycle  early 
in  the  process? 

•  ATEC’s  early  involvement  in  DT — Challenges 
to  ATEC’s  early  involvement  (DT)  need  to  be 
overcome  (for  example,  negative  OTA  perception 
by  the  PMs  and  increased  personnel  and  funding 
requirements).  How  does  ATEC  convince  PMs 
that  being  involved  early  will  help  the  process, 
not  hinder  it? 

Benefits/lessons  learned 

The  acquisition  community  needs  to  be  aware  that 
early  involvement  by  ATEC  leads  to  a  high  return  on 
investment  ROI  for  the  PM.  ATEC  has  proven  this  by 
having  success  after  success  with  acquisition  programs. 
Over  the  past  few  years,  ATEC  has  many  success 
stories  where  early  involvement  by  the  testing  organi¬ 
zation  has  saved  time  and  money  for  the  PM. 
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Figure  1.  M915A5:  In  an  effort  to  gain  early  operational 
feedback,  two  88M  Soldiers  accompanied  test  drivers  on  the 
cross-country  drive  from  Portland,  Oregon,  to  testing  grounds 
at  Aberdeen  Proving  Ground  (APG),  Maryland.  Each  M915A5 
(above)  was  instrumented  in  order  to  fully  utilize  all  mileage  in 
the  final  reliability  evaluation. 


Examples 

The  following  examples  of  ATEC’s  approach  to 
integrated  testing  provide  a  clear  understanding  of  the 
above  policy  statements: 

1.  The  M915A5  is  a  commercial  purchase  standard 
acquisition  of  a  line  haul  semi-tractor  with  some 
add-on  military  equipment  for  fittings  for  up- 
armoring  packages  (Figure  1).  The  M915A5 
T&E  program  will  start  with  a  cross-country 
drive  from  the  West  Coast  manufacturer  to 
Aberdeen  Proving  Ground,  Maryland,  on  the 
East  Coast.  DT  instrumentation  and  data 
collection  will  be  utilized  before  delivery  to  the 
DT  location.  During  the  DT,  the  reliability/ 
durability  testing  will  be  accomplished  to  the 
operational  mode  summary/mission  profile  usage 
with  a  mix  of  civilian  and  military  drivers.  There 
will  be  a  traditional  pure  OT,  but  reliability  data 
from  both  the  developmental  and  operational 
testing  will  be  used  in  the  evaluation  of  reliability 
and  durability,  thereby  allowing  a  reduction  of 
the  OT  miles  to  fewer  on  each  truck  than 
possibly  needed  to  address  reliability. 

2.  On  rapid  acquisition  programs,  in  comparison  to 
traditional  acquisitions,  ATEC  ordinarily  can 
only  get  a  very  limited  single  test  in  one  location. 
For  example,  the  initial  Mine  Resistant  Ambush 
Protected  (MRAP)  vehicle  programs  had  a  very 
limited  functional  checkout  as  opposed  to  a  full 
DT,  a  very  limited  user  operational  period  fit  in 
around  the  DT  schedule  and  a  parallel  set  of 
survivability  sets  (Figure  2).  Based  on  an  inte¬ 
grated  assessment  by  ATEC  and  U.S.  Marine 


I 


Figure  2.  Mine  Resistant  Ambush  Protected  (MRAP):  Military 
Subject  Matter  Experts  from  the  U.S.  Army,  Marine  Corps,  Air 
Force,  and  Navy  assisted  Army  Test  and  Evaluation  Command 
(A  TEC)  providing  ground  personnel  at  Aberdeen  Test  Center  (A  TC) 
in  generating  data  for  the  early  assessment  of  the  MRAP  vehicles. 

Corps  Operational  Test  and  Evaluation  Agency, 
MRAP  decision  makers  were  able  to  decide 
which  specific  variants  to  buy  and  field  in 
specific  quantities.  Based  on  continuing  cycles 
of  buying,  testing  in  both  DT-  and  OT-like 
environments,  as  well  as  Theater  usage  as 
reported  by  the  ATEC  Forward  Operating 
Assessment  team;  MRAPs  have  been  continually 
produced,  improved,  tested/evaluated,  fielded, 
and  had  field  feedback  leading  to  what  is  viewed 
as  a  success  story  in  quickly  increasing  Soldier 
survivability.  Of  course  this  quick  reaction 
approach  also  has  had  negative  consequences  in 
which  the  field  has  experienced  problems  that 
would  have  been  found  and  fixed  via  a  more 
standard  testing  approach.  These  realized  risks 
have  had  an  even  larger  impact  because  of  the 
many  variants  of  models  from  different  manu¬ 
facturers  that  have  been  fielded. 

3.  Certainly  the  most  integrated  T&E  program 
being  worked  by  ATEC  is  for  the  Ballistic 
Missile  Defense  System  (BMDS)  (Figures  3  and 
4).  The  BMDS  is  a  system-of-systems  made  up 
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Figure  3.  Ballistic  Missile  Defense  System  (BMDS):  U.S.  Army  Test  and  Evaluation  Command  (ATEC)  is  the  lead  of  a  multi-service 
Operational  Test  Agency  (OTA)  providing  an  Independent  operational  report  of  the  Integrated  BMDS  capability  to  defend  the  United 
States,  its  deployed  forces,  friends,  and  allies  against  ballistic  missiles  of  all  ranges  and  in  all  phases  of  flight.  (Approved  for  Public 
Release  2009  MDA  Book). 


of  developmental  programs,  upgraded  programs 
to  existing  programs,  and  integration  of 
existing  systems.  A  list  of  the  BMDS  currently 
fielded  systems  includes  Ground-Based  Mid¬ 
course  Defense  (GMD),  Aegis  BMD  Surveil¬ 
lance  and  Track  Destroyers,  and  Sea-Based  X- 
Band  (SBX)  radar.  Some  BMDS  systems 
fielded  for  defense  against  Regional/Theater 
ballistic  missions  are  Aegis  Engagement  Cruis¬ 
ers  and  Destroyers  with  Standard  Missile-3 
(SM-3)  interceptors  and  Patriot  PAC-3.  The 
Theater  High  Altitude  Area  Defense  system  is 
being  developed  and  tested  under  BMDS.  The 
BMDS  has  been  removed  from  the  normal 
acquisition  rules  by  an  act  of  Congress. 
However,  as  the  lead  OTA,  ATEC,  with 
support  of  the  other  involved  Service  OTAs, 
has  developed  a  fully  integrated  TScE  ap¬ 
proach,  which  is  making  use  of  collaborative 
testing  on  multiple  occasions.  The  BMDS  will 
be  fielded  in  capability  sets  and  each  capability 
set  has  a  planned  TScE  campaign  that  is  built 
from  pure  DT  events,  operationally  realistic 
events,  partial  OT  events  with  real  users  in  the 
loop,  pure  OT  ground  and  flight  test  periods. 


and  real  users  in  real  operational  periods 
leading  to  final  user  acceptance  as  a  mission 
capable  war  fighting  system. 

Summary 

An  integrated  or  combined  testing  approach  may 
offer  an  effective  means  of  shortening  the  time 
required  for  testing  and  may  achieve  cost  savings. 
However,  when  this  approach  is  used,  it  is  imperative 
that  extensive  coordination  is  done  well  in  advance  of 
the  event  in  order  to  ensure  all  (DT  and  OT) 
requirements  are  addressed.  The  utilization  of  the 
AST  and  the  T6cE  WIPT  can  help  with  this 
coordination  by  providing  mutual  support  and  sharing 
mutual  beneficial  data  across  the  acquisition  commu¬ 
nity.  Every  commodity  area,  every  system,  and  every 
system  phase  needs  a  tailored  T&E  program.  Every 
T8cE  program  ought  to  be  looked  at  to  see  if  the  best 
mix  of  separate,  integrated  and  collaborative  testing  has 
been  built  into  the  program.  ATEC  early  T&E 
planning  reviews  take  a  serious  look  at  each  program 
from  an  integrative  approach  to  T&E.  ATEC  lessons- 
learned  have  shown  the  value  of  this  approach.  We 
expect  to  make  further  advances  on  this  path  in  the 
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Figure  4.  Integrated  Developmental  Test  and  Operational  Test  (DT/OT). 


future.  As  discussed,  there  are  difficulties  in  fuUy 
achieving  integrated  testing.  However,  as  a  community 
of  acquisition  and  T6cE  experts,  there  is  no  doubt  we 
will  be  able  to  make  substantial  progress  toward  our 
ultimate  goal  of  providing  systems  to  our  soldiers  that 
are  proven  to  meet  requirements  within  established 
time  and  cost  constraints.  □ 
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